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^iU s ft?tl^^K^I>o Ufc*«oT, C I E L ABMW^ifflv>4 £t^-e§ 
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©t^U a **340 s b *;*^-;W4, -£tL-etL- 6 0 frtb + 6 0 £ IJ-C&So £ 

[0 0 5 7] 

^3 2 i:fiv>x, Wftfe* J: VWft*^ nft^UTj frftMftrs&ftK^ iar**LfcB 
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^T, /JnS fti'J T (*>fX:&«BMt* !> <b/h3v*.xy 7) iWfflO J: *)7C§£^y 7fc J: o 

^'J 7£7C§;&:fr<^y TCi^^tSo 7t^3 7, 3 8**0*3 9 ii 1 ®c??a®-e 
I«n;t^Tl4 0 L^L&tJ 5 ^ '7:/3 7, 3 8**0*3 9 *> jg LjSJfl L 
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x. sr -7fi?nj Qtc^s 0 yij P ^ «t 0«Q tt Lf ^ Jjt 
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>^£BEC*:£>t-fflv>f>ft& 0 

[0 0 7 8] 

$ ft, & t& „ 1® ^ t THKt is J: V lESHfc S ft & 0 
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J: »? *&v*ffi* 9 4 is <t Zf 9 6 £ <h:d s ~e§, ^^Ctti^i/h^fx 

T'S 4*5 J; 1*8 5 56 s ^f£>ft£o L^ ! oTXf7/84i:^t, 7jc*PM£#5e U J: 5 
<t1-^^<^7) s ^$ft, ^ft* ox ? LTEtt i * o ?^<BlS$ 

KIWLT r=^#mj ^it^fcCffiStSo 7f^8 5Civ>T rn^#jffi 

3 tcES-f-a :twi£ fl 

[ 0 0 8 0] 
[ 0 0 8 1 ] 

ti&AT-yfi o o-eB8*&i-*o ift»cJ: 4#5ecdt;^^ h Jt* s ffisfefe «t a*~*ift$ ft 

r^io2m) . ^ n -7 /ii* ^^cfigt i> fcfc t ft^a^iat 

^SofiKW (thin) 7X^^ h ikK^j-frT A *^L&^W&&%^<> 
[0 0 8 2 ] 

< h^l&^fr*mfc-t2>* h y y * 1 0 7 £H18 K^-To ^ D y ^ifcHJK*3ttij£© 1*1815 
(fractional) &«^7jC^[Rj t M^lRloH^-eMfe? fti>o #^fa<^7 JV^ W It, 
M8 icgffca-^r^ y^H^fefc^9-7^f>^A6So 2o<7)S[jJKp en a I ty», 
P.filtfp e n a 1 t y hBPt , *^8feHUc J: oTjfi^to-frio rfcfc*, 
max (penal ty V ert, penal ty h0 rtJ >0 . 75 "Ci£itl#Si 
penal ty=max (penal ty»„„ penal t y h ,rn) 
is X 

max (penal ty ver t, penal ty hcrf2 )=0. 75 "C£> 
penal ty=mean (penal ty ve rt, penal ty ho nz) 
[0 0 8 3] 
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4d }l 2) o 
[0 0 8 7] 

0 6BTW\ ^tJVf^ £#ccoJ: •? K^Mii ■*-*«, if\ n ? T^^v^-r i) i &M<DM 
u-y ^-rfrf- A i »J tt^^^i:ov>TtSl 0 8*ff^o * 

v^^lcoviTtS^fT^ o »otvi4S#, 7n-ft- h fi;*-r v ~f 1 0 0 
[0 0 8 8] 

7*8 3 i:^"-^/t7^ti 9 Soti/^i/^^ > ii. ^7" ? ^ 1 1 6 K& 

[0 0 8 9] 

HI 6 AfcJtfH 6 B©Wlfefcs& *M(i, fc1B<&^T ^sffi^'rv 
6*1* nW*&-fe?H £JBv>T 1 of fcttffiSfc^^Jgft^ n y zfmW&$;#>2>fc)£><D 

[0 0 9 0 ] 

I"* W»tJ 1-*tEi?Sr^-— ^£^x_*£fcKJ:oT^iE3*t#* 0 #m£^Kl~e 
^^#tti-*-^fi, *^5077^t'* (It*©*-* >X-f «fc CFM&JfrT* 

*>iS«*&»^n.y tf^^i-a*#, B«<s>**«*tt, ^ny^Bfcfc^nrfc-e*** 

[0 0 9 1 ] 

■^*£#5£1-* X 7 K%}Mfo2iL&& 0 -utli^ < t % 2o^j$M^T f ^OTfg-ea& 
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hj i3ir>^ r^^Kx^-^^j «ts 0 mi<om^.m (**-m--m i±, <k<omm 

Cfc-tc) / (T-JO =0. 7 5 

0&-;fe) / (T-±) = 1. 3 3 
[0 0 9 5 ] 

i2i9 *0m-r& t> ± «jaatanw*&*y Mi, 5c«>fljcija*B«o*^:«in 9 oft con 

fc&fl^^lHM^V 71 8 4, 1 8 5^n 7 

l^-TJ: 0 K s Sill (30, 1 8 0) ~ (2 0 0, 3 0 0) (3 5 0, 50) 

~ (6 0 0, 2 4 0) *H^2o©i^i«J71 8 4, 1 8 5 h HX^ £ & <D t fa 

5£1-&o £oMfcJ3V>T, B«^#(ii£Ji©ffi» (0, 0) £J: TOTO&SI (6 4 0, 
4 8 0) £*£o 0 
[0 0 9 6] 

Ml (£«©H^xV7 18 4^*) ii, 4fe^)WI&t J:oTJfejt$*i4o 
£>0 £<3 0 

Ji>0 _h< 1 8 0 

1a>2 0 0 fi< 3 5 0 
T>3 0 0 T< 4 8 0 

M$iW2 ffi«oM^i'J7l8 5©*) (i, 3fe«)WftCJ:oTfei5S*L*o 
£>2 0 0 3 5 0 

Ji> 0 J;< 5 0 

£>6 0 0 fi<6 4 0 
T> 2 4 0 T< 4 8 0 

(W*OM^U 7**frtp (18 6)) Ji, 
0 i<3 0 

±> 0 _h< 5 0 

^> 6 0 0 fc< 6 4 0 
T>3 0 0 T< 4 8 0 
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[0 0 9 7] 

Wife-fey h<0|5£iftr*Jlv*t\ #K<0ffl£ifclfcfl94jKRIHry f^Itifc^lSo *M 
Jb3 ^co 1 „ M^^tOEiS (4o<7? [3fr<Dl] ©#M<9i:i?), 

[ 0 0 9 8] 

^fet~> yp°nS^ h V <y ? tat^-cfrOTttft^flSlftlEKIfiH^fe&iW 
[0 0 9 9] 

±fc$itffc«tt£T r^-K4J Wife-?* So «;to*>> *fmiSefc£*tfetffcP,1% * 

t^KtSii:, 1" * *> -fc»W*»Jfr*3»ff * i5 4^ £ £ ^ i- frr- 4 * »i" £ t *« 

§I£ Lv^o —mk LT, TMFi&oiEiig* l©9'f>^Eili, iEffS^ 3 l o 

[0 10 0] 

x = 1 / 3 

x+el-e2=l/3 

penalty = clel+c2e'2 + 

[0101] 

^7 7^2 0 6 11, ^toW#fJ^JSa*#t>*t LT, #W*&*f lof 
[0102] 

"7*2 0 8 fci3V»T, Sl£<&W#bfc*3t**n? Ztft^SiH^lSo 

[0 1 0 3] 
[0 10 4] 

CSTIiii.Wittf, 2-=><^>|«MA*J:^B (AttBSXfefcKifc*) i, At**© 
3fr<D 1 ©v^f B££1R0 3 3'e>ie>?'r>#3c£S1tLJ:3 £TfcW&e>iSa* 

-frta-fri:*****^ Wfc^fcJSfcfcfrfeLfc^o »<^SW*&-fc^ FtMtl.^T7 7 

[0 10 5] 
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[0106] 

[0 10 7] 
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tz it V 7 Kfl:3fr*Ste\ Xtv72 1 0 tJROint^Tli, 

fife • ^^^coJi^Jg^fb^Pi^i Ltffiv^fJo 
[0109] 

< s^-r % t , ^<?> x y v <jd s it p m&w & r « -r-e & 5 5 s * + <?>#m * Bfc^-r & £ t 

[0 110] 

n<VW-m%: (boring) ««^ffl»lffii|H|W«LfcW«lT*4o P*>=Hc, llli*^ 
ds J: cm K^lRl «9 ? n y 7°&S K L tz £ oife trr t oftH^ i i "C* * 

°[0 1 1 1] 
[0 112] 

1 2^Pj, ^J#Pfi. M*b*fey h©3fi*'&'fo** , i-'<Tift*L*:**ST5*to^T 
^2£fr9 ';7"2 1 4 IC&&-0 tSLT^iv^ 7n-f^-M^rry2 0 

o 01 0li-ffJ(?)*^L, £<z>f*TNi, ^-r y T 5 ^ l 6 \z&\,->X?n yym^^^±^ 
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fy^2 1 8 K3s^Tz/X-r2»V>3.--*fKM7F-?&o 
[0 113] 

tJB^j^'b Lttftv*. £*lhu <-T&J ifcfi l"*3<i-*J iv>9feHI|iS:37 

t % ji- if *«JE#fi L fc MURM. t K Tfr tz 4 * n 9 J: fe^i^n 

$ * n * yaufffc 4*** a y -f$m k%y), MIRZ fife ? n y ^co^-f X £±@ 6 

10 114] 

[0 115] 
[0116] 

HI 1 1 (* % ftit«IW!i^- K*$HiB1M8jfc Ltlg? *M*!it*fK^n y 7° 
[0117] 

Jfc« Ml 0 <T>7.-TV ~fKMf&-r&m 1 1 <D7>TV 7Ki$ 1 0 Otfftf 

[0 118] 

ffitiltiiUft-fcy h^ofc^iii (3 0 2) . £*Lfe*9IJ*L (3 0 4) , 
— W<DM$3-ty h% Wmmfo (driving constraint) -fey hj £ LTMtS (3 0 5 

[0 119] 

[0120] 

5 

£=38 
2fe= 15 0 
[0121] 

[0 12 2] 

If) -e&& 0 

[0 12 3] 

? rJtlW*&-fey h-e*»?#So 

[0 12 4] 

iWWWife'fey Mi, :/IBB£>3 *><DFK* r^Bfei-ftJ ****** 
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[0 12 5] 

XT?? 3 0 6 *WWI&<B*<Ofi*^!b* fc LT, #W*MflHKHr ? ^fefflifc 

[0 12 6] 

[0127] 

[0 12 8] 
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[0 12 9] 
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[0 13 0] 

o 0 1 1 (4— ' W<0**^ U ^co^flTJiv Xv-y/S 1 6t:i^T^ny7'ti^^tJl' 

^■r '/73 18 Ki$v»T5/^T-A^)J.— if^SI^-?-£ 0 
[0131] 

[0 13 2] 
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[0137] 

[0 13 8] 
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[16 A] HI 2 A & J: CF0 2 B K^Lfc <t "5 4*^Si:J:SM^*«tl>-^*ffI 
[El 6 B] HI 2 k&£ZfW2BKm\^tzl 7 ft&^BfUz J; jfrifefcfflMr 
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The present tovejition reLates to the automated and scmiauLDmatcd capping of 
eleclrCHttc images,, arid in. particular to an apparatus and a method of using an electronic 
5 camera to oajrtuii: iiadorops-jclieicctrooip hnacttt. 

Conventional photogmphy requires the photographer to use a viewfi&der both to, aim 
tins camera and to compose the picture. Composition fit a psrttottlar location b done by 
charjgbig the direction of the camera cud altering the zoom control Careful 

10 composition fakeg time and attention, as well as an understanding of Various ndca of 
good photographic composition. This is a sJdlE thai many find hard to fcaiu. The «CEbit 
required causes flic photographer to be "out of the even*? and in Many wste ibis 
pcofclttn » enough to prevail potepfiaity pleanng photographs fiiSm bemg tekea. Thfe 
a particailarly the case wbeft a photograph has to betaken quickly, for example -tffrca 

15 photngraphiiig uctiun events o* chiltfmi. Although, in principle, t phoJograph est be 
cropped sflcrtbe evcaJ : this is thrus coaamjilpg and inconvenient, and may Kttt! require 
tojowJedge OftlW roles of good photographic composition, which the photographer may 
nos possess. 

20 LIS 5,97^,519 discloses an automatic cropping, apparatus for croppmg vetf specific 
images, and in particular "portrait" photographs where an individual ia positioned in 
frwiJ of a uniform background. Under these relatively welt defined eoadUJons the 
system disclosed in US 5,978,519 firstly converts the tormt image from ROB colour 
space into luminance space. The image is thai scaled so fit a grid of 256 x 2Sfi pixels 

25 which itself is sutflividcd into- blocks of 4 it 4 pixels. 

Next, the mean aart variance of foe ioteoauy level is oaloatawd for each block of pixels 
and a variance profile of the imago is calculated- Because the Iwckgroiitut is plain, it 
can be expected mat most blocks within the image will exhibit lMe. variance. A 
30 tJuasfoold ia calculated from flic variance- curve, the threshold correKpnndmfi to fee 
'•knee" of the curve. All Weeks \vhwfc are of interest are then cropped by bonadms 
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them by a rectangle which h seiecled as the minimum size of rectangle (and hence 
crap) into which the regions of mleres* fit to one embodiment of the invcjtfhrm 
disclosed in '51 9 this mkbmun Crop is thea iacrcased by 1% to define a Ixwdtf. 

5 A post procefsing procedure- may tfaea be performed to remove the *tefjise" within the 
ciop boundary, lias, as described in '519, "small gT itches tod spots are eliminated, 
[hereby providing a better bounding rectangle, especially at Hbe edges of the 

10 Thus, alOioo^ this prior an system do«i perfbnn ftufiwHsfc cropping of Images, it does 
so only on image* whete it tea a prior knowledge that the backgrouad is plain and on 
the aasiTOplkn. flieS fteae is oidy out? subject - nod hence it always generates a. crop 
tonndary to frame the one Bubje^ This tetfuikiTis is not ratify suited fa '»rd world" 
images where more complex scenes are cap&jwd, whore there rany be one or Bare 

15 oompeiing subjects or where, to obtain an ECsfrelfcally pleasing result n csjp larger 
WMiheminirruim crop able to ferae flwaubj^ may be re^rod. 

■A ccrurding to a first aspect of the present irrvention, there ia uroYKfed bu deetamic 
image processing device for cropping, an electronic image, compmuag an image 
2D processor, whereinthe device fa adfipGed to: 

a) process the electronic- image to identity one or more fenturt* relevant to the 
cmnpejation of the electronic image k each such feature occupying a sab-«#ozj of the 
elecjromc image; 

25 

b) select al least one composttiorial rata from a plurality of predrfremiiied 
compositional roles, based on the relevance of the CMrxposKiomd iute{s) to one or more 
of the identified features; and 

30 c) determine one or more suitable crop boundaries by applying one or more of (he 
selected compositional rules. 
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It is thus possible In provide a more convenient uppateftbw and mciiiod for capturing and 
cropping elcclKmJc irangesv 

5 Also according to the iavcntioa, there is provided a method of using- an electronic 
jjntige processing device for ctoppiujj an electronic: image* the image processing device 
coxtptMng an imago processing jewwuss, the image processing means hi chiding an 
electronic processor and firmware and/or software for processing the dcofroatc ansge, 
where! a Ihe method comprises the steps of using the image processing means to: 

JO 

i) process the eJettronic image to Identify oik «r mme feature* relevant io the 
composiiiaa of fee tiectranic image each such festate occupying a euh-regtoii of the 
electronic an age; . 

15 it) select one or ton re caqiposftional rules fiam a set -conlanrins a plurality of 
predetermined composkmnal rules, hosed on the rdevanpc of Lbc DrtmpoSitiooal mlc(s) 
to one or more of the identified ieataes; and ■■ 

til} determine opo or biojd suitable crop boundaries by implying tine or more fifths 
20 selected wmposftlona} rules. 

Preferably each identified feature has one pr more ccmposrtiooally significant 
properties from amougEt ji phnnlity of dxrtsieal ^vJcfc trn 1 nftri Compositional 
properties. Advantageously one or more compositional nflss are selected from a. set of 
25 predetermined compositional rales, based ©a lht> relevance of the cairpositioaai mJe(s) 
to the caaqxrafiorally sigftifi«ii« properties of one ox more of the identified fixtures. 

lite image may be cropped automatically by die image processing means according to a 
measure of the quality of the potential crop boundaries AI tentative^ a plurality of 
30 automatically crfculotod croppies may be presented to a user, and the user may 
manually select amoo'p>t the potential crop boundaries, so that the cropping of the 
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imajra itt performed scmiairfcmMJiealiy- 

ComvasitkiimWy siEniGcant properties of a feature htftlud* things like the type of 
feature, e,g- a blank area,, an ansa, of relatively high contrast or colour or texture, 
5 recognisable -objects such as a face, the sky cr an edge or the horizon. Thus, a "feature 
of compositionai significance" in the image may be me absence of ho object iu ft 
portion of <he image, - in a blank area. 

The oicetrordc imaging system may be part of bu electronic carneca, or u document 
10 imaging system, or any other imago capture aysian when) the cap luted image may be 
cropped 

Hie electronic imaging system may be used with aa electronic camera system for 
Btptating as olectraraic image of a scene. En particular, the eJcctrouk imaging system 
15 roay he irLcxErpomtsd with ?he elc<xroDxe camera. 

Artcmarivcly, the system may comprise a coavontlnnal elerfronic camera that ourpins 
cqj>sbpccJ Sntagca to an imago processing device, Lot escRmplc a personal computer, or 
oiha data processing device, that includes the image processing mesne. 

20 

- Tbtf earner* will, in general, comprise a detector array for capturing the electronic 
image and an optical imaging Rystetn arranged to image the scene onto the detector 
away- The camera may be a hanJ-hski still electronic camera and/or a video electronic 
camera, 

25 

A first con ijxwiUo nal role may comprise edge placement criteria, for example having a 
dark border (o an cd&c of th^crijpped hnAgc. A ttyihir cttrnposidorifil n)Jc may ssCfcto 
place an edge one-third or two-thirds the way from an area of interest within the crop 
hffvaidscry. 

30 

It may bfc (hat just one compnsStional rale h used multiple times on. a single image, 
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once fni each idcntt£e6 feature 

Ojscb candidate crop botuidaries have best* deicrfuined, these ri;ay to presented to a 
\jSUT Of thCf sysbap. the wssa- may then choose 8 Cropping candidate, whCrorpMi the 

5 image precising itovice muy bo futm&A crop 1h& eJcctaoofo snagc ecsazding tu ihw 
user's chcac*. 



The identifiable features should be those features That ere relcvaot so ibft placement of 
cropping bomdnrioa in the processed imago. 

JO 

The camera may inciudo means by which a user of the camera may la* one or more 
restores relevant to the composition, of the electronic image, said tagged teamcf s) tf&a 
bang associated with a cOmpOSitkmaJ rule mat include* said tanged features) ia the 
cropped image. Such ta^giug jnay bo done by Cho user tu*ltca&t£ by roams c£Kiitfib!e 
13 controls a future or an ar ta of the captured inaage as di fipteyorf to rbe us**, for exmrtpte 
on a LCD di splay buiti mto the camera. 

It may be possible, however, for featoew in t&e inrageio be tagged rimom &tjcaSIy, For 
exampk^ a person TO view of &e earner may wear some sort of identifier tag which can 
20 bs recognised autnmaticHlfy by the image processing means wirhin the Camera. The lag 
may be m optically ctfsringniEhabb badge jk&ccr understood by the imapn pm ceasing 
software. A pcraoa can then be antornafceally identified. 

Optionally therdfore, the idectifkblc features may mthido a prodatynriiowl feature*, for 
25 jexsmple a tag that a person may wear. Such a tag may have an idcedrlahle pattern 
Which tf>= image proofissing jsystecn recognises. At leasi one compatntional rude wfll 
thai be ttSBOciated with such, an identified rag ao tiiat die jdenrificd feature is included in 
the cropped image. So mat the ta& does not distract a tiwr viewing the crapped image, it 
is preferable if it is. implemented as an iufia-ied only teg (far cxarnpte, as an infra-red 
30 transmitter). 
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One way of manually tnggjErg features is to use the camera viewfinder as & pointing 
device (as opposed Jo its conventional use as holh a poiutjog and coajpeaiuoa dftvico). 
in this use the prime area of interest is dattbexafaly pjsitloued in. the u-pproximEte ccatre 
of die frame. When the troage is auttHcstogped accortfiag to this invention, die region At 
5 die centre of fee maac is deemed to be essential to ihe cropped imiige and is thits 
prevented £bcnrn being cropped out. 

Eo another caibadfmBat of the fnvcofjon, pries to idenbfyirig h\ the captured nna£c any 
features, the electronic camera may be used to capture an image of object with an 

]0 appearaace cormspoading to a feature to bo iaofuded ia iheatippcd bioge. This may 
relate- tn (be appearance of the olgect. (e# tfw person -for cxampie,' a face) itself, bat in 
a different use model may also relate to appendages worn for- the purpose of 
identificatWKt. Par example, if a panscax is wearing a jacket ofa pardcitf ar shadoof blue, 
then the electronic camera may be pointed at tiu jacket in orcfer to TuBbEst? feo image 

15 processing means to recognise Ift&t abode of trine as being aasodated -with a togged 
feature ^ m that aharJe of blue ia eoptand in an decttome Image, TSua may be 
flssigued a bigb itttcteat metric and.^rmay be associated with a particular compa shionajf 
tide. When an image is captured, the imago proteasing means may tbrti be used to 
jdtjriifv hi the coplrned image at ioast -boo tagged feature. Then, the compositional rules 

20 ty sy be used go crop the captured image so that the tagged feature is mcluded in the 
cropped im&ge. 

Tn tine «?rnbu>Q"iEicfit of the uuveution, t&tsp iii) comprises the steps oii 

25 iv) gtraerating n piurahty <^allCTnaJiv&cradidair;crop boundqries; 

V) generating a measure of carripoaiticm quality for each ahem stive candidate crop 
boundary by rains a «t of picture compasition metrics; and 

30 vi) 8 electing s» an output a reduced number of crop candidates having a relatively high 
measure of composition quality, for example, just one crop caridTdofe. 
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Iq aaahernHiivc embodiment of the invention, step eonipriiws the steps of: 

vii) implementing the compositional rides ns constraints ibat may ho satisfied to a 
5 greater or lesser degree, ' each, constraint having nn. associated cojjt fhncUou that 

increases tfce leas weJf the constraint is satisfied. 

viii) deiitrin£ an overall cost metric aa a Ftmctian of crop cMidirurles in foe image, by 
fodnuig a combination of the separate cost functions associated with each icdividuaJ 

10 cocstoirii; 

is) applying an optimisation method to rind arte or own? best crop locations by finding 
minima in the crv<xall coat metric; and 

15 x) selecting as output a. reduced number of crap candidates for said test crop l&c^rts. 
forBxernplesjostooe crop candid ate. 

There -will be at least oao crop edge. For csMae^eacixIeoTandlhjsso has just oneenop 
edga. Often there will be more than orx crop edge, for exunple a square or a reetanfflb 
20 hna four ed^es, between one and four nF which will result from cropping af the original 
electronic image. 

When the crop boundary of the cropped Image ha? a plurality of crop edges, t^e steps of 
the/ method esay be perfcrraed acparatrdy for each crop edge to generate the crap 

25 boundary, 'frus kelps to reduce tlw cfilraifclion necde<1 1o .select foe ernp boundaries, ai* 
each edge may be assessed indepesd -truly frtun flte other edges; However, If the user 
seeks to apply some aspect ratio consrxair.ls then the ammint of oamrRrtatkrn may be 
reduced: Thus, if for example, the usqr defines thai the aspect ratio will he exactly A;B F 
where A and B are n timbers, such as 4 and 3, then once three nop be lindane* are 

30 known then foe position of the final boundary is fixed. 
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One way in vt+dcb the invention may be implemented is if step (J) described above 
includes ihe gfcp of identifying fixtures u-hrcli constitute noa- overlapping $i^mcnt«J 
regions interest. T3icn step (ai) may be achieved by first selecting alternative 
divisions of said identified regions of interest into those which will be included by the 
5 cmp bonadary and those which wia be excluded by die "Top bowKhlary. Each such 
alternative division of regions of interest is then uxed to determine an inner crop 
boundary Ihnwt and an outer crop boundary limit such that the inner crop boundary limit 
is the swrftest boumdary which cucumwrifocsii the regions of interest m be included and 
the outer crop boundary liartt fe the largest boundary "wfhich excludes the regions of 
10 interest to be excluded. The one or more suitable crop boundaries can then be 
determined bcnve£.D 5he Ihnitfi of the inner and outer crop boundary limits for each of 
sztd dtwnaiive divisions of regions of interest 

When the crop boundary has a plurality of edges and one or more adtnSie crop 
15 boundaries Rro detrained by evaluating &n edge qimEry matric for each possible edsc 
plaeeascnt between Sue imw anil oufci crop fcourKtaiy limits, 6W selection of best crop 
boundary may ai {cast in part bo tfcpcntfcmt on the BCpaxatc edge quality metrics of each 
of the bftundftr/s edges, 

20 This can also help to reduce the computational burdcti, pa h is then not necessary to 
consider altera iitivu crops where the outer, crop UirKyiaxy limit does not ccrapletsly 
enclose the inner crop boundary limit, or whae the s^aipe. cf the area, between tSio outer 
and inner crop boundary hunts does not penrdt the placement of the desired crop 
boundary shape, 

25 

ft may he that In stop (i) blank or-unuteresring areas are detected as some of the 
features teJevast to the ooruposition and hi steps <ii> and <iii) one or more of file 
compositional rules causs the image to be estopped according to the arrangement of said 
blent or Unirdtsiestirig area* rehtfvc n> ofter identified features relevant to the 
30 composition of the electronic image. 
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ltt photography, audi blank areas are usastily associated with a plain bacfcgroiiiid, or sky. 
The image processing raeans can use one or intae compositioxia! rides associated with 
soch blank For example, & blank area exfeudinz exrtois an upper horbsaitfai Siand 
of the scene may be associated with plain sky; and so aa appropriate oorapositioiial rule 
5 might b« to mmiiffiB the amount of tfcia particular, ansa, and lo orient hcuhyin tally the 
bijuoilaiy between this area and lower areas of interest. Therefore, in general, the 
method may comprise the step of using the conapoaitioiiaL ruJw to crop the captured 
image according to tbo amangemeni of tins blank areas relative to the other features 
which ere not H sok. 

10 

If flje imsge is ^ odour image, ii may bo &Bt at least one feature is identified by 
segateridng the in»gc into areas of similar cohyor acd/or lexmre. 

A profaned way of jtfeccssius the im.R£& to identify ih» vr each feature relevant to the 
15 eotu^wsidoji of theimG^cOJEprises: 

1} optionally resampling the image to a reduced number of pJxeis; 

2) bJurfins tbo images; and 

3) mergifig regions of shastar appcaearice, 

SO Tho blurring of tSue fraagfrhafi die effect Of removing fine detail, thereby ensuring that 
insignificant areas of nigh contrast, for example si? ihnragh loliage, do not get given 
Undue aUeolion. . 

The ''region TDKHgC' then groups adjacent sections of the image having similar 
25 appearance, for example similar colour, together. The merges 19 advieiUigcoualy 
performed itteraiivoly end a teg may be performed at each Lttcmion Ui ideality the 
numb&r of different regions within the image, 

Aa part of This process gsnali area* of colour scaounded by larger ureas of similar 
30 colour ot intensity ntuy «5so be merged into the krgLT area. 
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Colours become clustered daring the region merge. However, d ftnther cokrjr 
clu.-5ttd og step mny be performed to reduce 'he Bomber of colours. 

S Tite tiansualttcw of the toJour within the image may thru be adcvlatcd and can !>c esed 
to derive « "aalkncy ttnags" vraidb. ituscatiis how rignificanf regions of the original 
image are. The salieccy image repreaeais an interest metric for regions of the iraege, 

The term "Interest [aeirjc' 1 is used herein to define a weighting or unpnttancc attached 
10 to C particular area of Interest. The i uteres raielric fiif an. ideoiiSssd feiihife rasy bo used 
to associate with Otis feature at least one paxtknriar photographic, composittoaal rale, 
For example a feature with a high, interest juetric may be associated with a rale that 
piacea soch an area either et the centre of a cropped image, or slightly tp one side of the 
centre of an image, for example eccQodhig to the well-known phoitagfnphic 
1 5 comrposifloiial ratio called "the role of thirds*. 

AdvwtegcoBsJy the ^yfltem, may also- be capable of ftee detection, or person detection 
using; one or roorc of: ctolhi tig recognition, Cvce detection, h^r cn-Soox and approxiuiqte 
style recognition., or use of body shspe Jtrmdclft, 

70 

Certain regions may be denoted as more saiieat tSma othera by allocating htgh aa&Bice 
to segtowi Ob the basis of £ rone combination, of 

- rebti ve d nusuairx'as of the colour, inlstiaity or tcxi aFe oflhe region to other adjacent 
25 itgicna; find/or 

- relative umiaialness of the colour^ inionsity or texmw of (he tegioti relative to a 
satatarrtM portion of ike it&a«c, 

30 A decision ewy then be made by the image processor, software and/or firmware a* to 
how likely Hie region is to he a resorts face, tend or whole body, or how likely the 
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reg ion t3 to be a known pereon or how cento! the fagicm is ia the Ullage. 

T3ie system may- also allow for interactive inpiU from the met trt indkate the prime 
region of interijst, 

5 

The uxirposiiicraal role for features identified at the centre of Hie image £0317 be to 
include such feature i in the cropped iirmge. 

The invention will now be described by way of example only, with reference to the 
lfl Accompanying drawings, in whicb.- 

Hgure 1 is a block schematic y*ew of sn eledraaic canwaa system incorporating an 
electronic image processing device accord mg TO 1ha inVCrUioOj having, a detector array 
and an image processor fer antomsticaLIy cropping electronic images captured by the 
J 5 detector array; 

Figure 2A and 2fl are flow charts illustrating fen tmhodiment of a method according to 
the bveaiion lot cropping (1 enpturcd electronic imago; 

20 Figure 3A in 3F shaW the processunr steps obtained by following the method described 
wrtb reference to Figurea 2Aani 2B. 

Figure 4 shows a cumulative histogram of coksured pixels agahiEl increasing colour 
distance JroHa a colour cluster under coosirlcrarion; 

25 

Figure 5A to 5C illustrate possible crop bouaaeries; 

Figures 6A end 6~B are flow charts showing in detail one way of inip!e:oootiug the 
mtfh<4 aecordrug. to {he kiveojjon as kpowi in Figures 2A arid 2B; 

30 

Fhpire 7 is a plot of crop penalty metric against crop distance; 
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FJgUffl 6ba p?ot of penalty metric versus position for ilfetnarivft positions of a main 
region of hii£*c5ti 

5" Figure 9 shows ischcmaticaliy mixmivun and maximum cropping rectangles for two 
identified features; 

Figure 10 Is a flow chart Viewing in detail one way of implementing the method 
jwwmrmg to the invention as shown m figures 2A end 2B, wdog a "constraint" fcased. 
10 epprotuA; 

figure it is a flow chart showing In detail one "way of inqaJemcatlng the method, 
according to Uu invention aa shown in Firama 2A and 2B, using a "generate and test" 

Figure 1 2a and 32b $?icrw examples of «tn airtamaticalry r>-ndtarai cropped linages. 

Uigroe 1 Rbmvf) sjchematically aridcctirjroc camera 3 far capturing an decrrccric image 
nfasoeoe 2. The cwj&c* haa an optical imaging system 3 arranged to imago tile «aa»2 
20 onto a- two-dinoens-onal dtsfcwtar array 4 wWho. tf» etsraa. The defector away is 
connected 5 to a waicrornxtfsesMc 6 Eat processing of the hnagca captured by the detector 
anay. The Btkiopstoccssor 6 has interna! software rajd/ar firmware for processing 
captured electronic imag es. 

25 The microprocessor 6 la itsdf connected to an im^ cqttTe button 7 by v?hich a user 
of rite camera 1 may initiate orpfnro of the image, and also to a mwsory S. Not shown 
are various items normally associated with a conventional cJoctiOcnc camera, namely a 
battery power suppry, viewfinder or liquid crystal viewfinder display, focus and light 
level detection optics and fihxrtronics> and exposure and auio-fbeus com ret 

39 mechuafcra. 
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Preferably, the pj*>eesssog of the captured im^ge 10 e perfocssed by fee ndocoproceasar 
6 in if» caroeta 1, witb the Lriprumy 8 serving to hold date genera Jed by the automatic 
cropping prtOccsety and tite fesultaflt .cropped ftuages 59. Tl wooliL. however, be possible 
fat image processing to be done external to a camera body, in which, cast: flic ctecironic 
5 camera and exteiml processing form the electronic camera, system of the inv.miioiL 

Ihe invention '& particularly useful in (he case of electronic cameras having a detector 
array A with n relatively large n amber of detector elements. For example, a detector 
amy hirving WomflUon or more elements can be used with an Optical system 3 having 

10 a wide angle fictd of view. The user then, need only pawl the electronic camera 1 
generally in the direction of ii scene 2 he wishes to photo gfpfii. Automatic croppbsg con 
then be used as described above to crop tuiwsnted areas of the eaptttted image This 
zcHera ft considerable burden from the puotogEiplKr, as be no longer needs to worry 
uriduly about drfaife of pfcotog^Ke canmnsitioo. EkctronLc photographs can then ha 

15 taken rapidly, which increases the chances that the. pliotogiapbcr will capture the 
desired moment. 

An CDQMrpIO of ft jnettwd used foe processing wiagts in accordance with one 
embodiment of the present invention will be descritrd with reference to Figures 2A and 
20 2h, Hie various steps of the method, together wjth their results, can be seen with 
reference to flgtncs 1A to 3F- 

Figurc 3 A shows usage composed in a casual manner* perhaps by an inexperienced 
photographsx or by someone who web rushing lo try aud capture the scene. The picture, 

25 generally indicated 10, shows a gzzi 12 on a swing- The gid 12 is reasonably well 
centred in the Image Bad can be assumed to represent the Trtfr'n subject of the imngc. 
However, other potential regions of interest in' the image include the flowers 14 located 
towards the bottom left hand comer of the image. Qa further inspection of the image it 
can be seen that a serious ajmpotfttonal eiror has been made by die indusfon of part of 

3G the figure of a petflon towards the right hand side of the image, and gencmlly jndia^ 
16. 
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Au automated Image processing system taw no a priori knowledge of the subject matter 
of the photograph and tiiercfore needs to process ii in order to extract seme £arm of - 
teprtaeatatioo wbfcii will indicate where the compositic»a3ly dgnificanl regions of the 
5 photograph lio- 

Thc photograph 10 may have been taken wilh a camera having In, oaxss of -2,000^60 
active pixels. Analysing auch a. largo ntirnbcr of pixels WO^W be compute tianitlly vary 
sinificant indeed. Thus prior to performing any other processing staapg* the image 

10 pioctssor down samples the image in order to induce (he number of pixels therein. 
Figaro 3g sebenwticalry illustrates the same image as shown in Figure 3A, but aftw 
doTO samplingfio 240 by ISOpbcsis. This down sampling has reduced tbe lumber of 
active pixels to 43,200. Pall owing the down sampling, the dou^aaiftptod imago 18 is 
thea converted at step 30 into an image having compressed ootoor vatte&ori Vilulst £rtill 

15 rets&dng intensity variations. An example of such a peaces sifts is coinrcrt&g flic image 
to tho YCC colour apace rnrnuL tl sltould be noted that this is not lite only colaur 
space lepmnfetion v,mch could be nscd. Thus, dm CIELAB colour space system can 
also be used. This aystara is well known, and deSnea a epace in which tho lightness L* f 
wWch H a measure of haw height a colour is plotted against tbe vertical axis, and two 

20 finiher incHsaremedta a* and h* are defined as linear axes with a* defining {be colour 
from a red to green scale and the b* axis indicating colour on a blue to yellow scale. 
The me&sittejoaenrs o* and h* arc in. the hoii zoriial coloar plane aod are perpendicular to 
each other mich that this colour system defines an orthogonal caitesMui space, Each of 
tho L\ a* and b* axis are defined ta such a way that one unit on any of &c scales has 

25 approximately tbe same "viaibiUty" ww&inE this system both lines* aad fcottepic as 
regards human peremption. The L* axis has ft stale from zero (black) to 100 (white) 
whilst the a* and b* scales range from -£0 to +60 each. This system has the advantage 
that a colour difference of one wait has aubsfiwtially the same visits Siity at any part of 
ihe colour space. 

30 . 

Following conversion of tho image to a Colour space, areas within the converted image 
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having similar cotoer and intensity are generated and grown. Thia process cormneiwes 
et step M vjhejetbc image ts biurrcd, and then tic b'mrcd image is analysed at step 3-2 
in Or4cr to foim "seed areas* tint have a smooth colour acd intensity. The .-*ecd areas 
are then grown fit step 33 by adding areas adjacent to -the boundary of the seed areas 
5 wltere those adjacent areas have a suflicisoi!}- sarrulur colour and intensity. Froni step 
33, a rest is made at step 24 to de{«mine whstber all or' the pixels within the oolonr 
comprcssed image hme "been allocated to xtrfi oroai. "if TKrt, thau corJrol is parsed from 
step 34 bask to step 3 1 and the blur and region; grow process is repeated to an iterative 
manner. 

ID 

Eventually, the test at step 34 will be satisfied. Figure 3C schematically illustrates the 
irntgo SB once ail of the image has been blurred and assigned to regions. At this sfage 
the Linage shown in Figaro 3C contains approximately 2,500 region^ some 3,200 of 
which contain 10 i>r leas pixels. 

15 

the image jwoeofislfatg ttoci continues at step 3? by mcrgttg adjacent areas oftheimaae 
wfckh are separated by Sveat edges*. - B Weafced^es p are those boacdaries^iat separate 
areas of the pictufe wide* have a relatively low coburot totoasity differaiic**. In other 
word£, the regions «re close to one anolhot V^thm the YCC or CIELAB apace. Photo 

20 step 37, conteal pnwd to step 33 where Adjacent areai with similar ^tjtMm ecloura are 
merged together. Pram step 38, control is them passed to step 39 which, examhua the 
linage to determine if small areas, that is arecs -whose size & less than atoreshold value, 
arc ceaopletely enclosed by another larger area. If so, then tEae Email srea is merged into 
the larger area. Steps 37, 3S and 39 can be applied in a single pass, However, steps 37,. 

25 38 and 39 may he applied htsrativety and a test may he made following .step 39 to 
determine whether ma number of individual regions has fallen to below a 
u*edetehnined threshold number. If it is judged that there are still too rmwy regions, 
then atepa 37, 38 and. 39 can ho repeated, possibly wflh lha dotmitiffin of what 
constitutes a weak edge being changed such that tfw dialaatoe in the coloar apace by 

30 ^hicli colours most be separated beforti they are regarded as sufficiently ojttwmt coc to 
be merged may be mete&sed. Figure 3D shows ti« image following the region, merging. 
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From step 39, control is passed to step 50, in, Figure 23. -where Ihe imager hi further 
analysed m order to elastof similar colons together itatil snch time as lh<s n««*cr of 
coloDtfl has dropped 50 an appropriate number, which is typically in lie region of 20 or 
5 so. The image of chetered colours is schematically iEnatetfed in Figure m. 

It should be aoicd that as used beraa s nogioa is & spatially connected sab-qtea of lite 
DiiaBfli. llovrcvuT a duster is a cnllr^ljon of rimilar region*, but tiie icgtws do not need 
to be adjacent to tme another. 

10 

It raa be seen with reference to Fi£tirc 3E that the main part of the Dowere 14 Low 
been merged hwo areas of uoifii:iu colour. Similarly, Ihe girl's face- bns been merged 
into art area of torhbrm cofow 51 as have her trousers 52. tatgc areas of the 
background have also been merged is&t areas of substantially pn&broa colour, for 
15 example the trce toward 3 the left hand aide of the- image. From step 50, cdntroJ is 
Initially passed to step 54 where a» interest metric is formed on the basis of the 
ittsaauakoas of the colour, and from there control cf passed to step 55 where ttue image 
is analysed tn detarmme the' cwxrpositiMjaliy significant properties Ui from 
amongst a plurality of different possible properties. 

20 

One such analysis tfmi may be psrfonmcd is the analysis of fi» clustered colocrs shown 
m VlskiTC 3E to determine how nrmsuat they are. Ihe image ahown in Figaro 3E> as 
noted tardobefcie, comprises approximately 20 or so different colour chrater^ These 
clusters are then sorted lit order to identify h/rw many pnwfe belong to each one of the 

25 00&HH3. 

risute 4 wltofnaticelry illustrates a cumulative iustogtaai of the percentage of pixels 
belonging to a colour against colour distance. Each of the colour clusters is processed la 
turn. Wlien. a colour t£ processed, the colour distasco between it and each of the olhor 
39 colour elaset* » calculated, il«j chistecK are then sorted in order of colour Distance from, 
the colour cluster being process sd, A. Cumulative histogram can then be formed for the 
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colour cluster tmdw test, by counting fre cqaraSative turn of image pixels which, are 
Included in art increasing roOTber of cfcisfcRS along thi colour distance d tmcnaion. 

Clusters -which, togcthar with closely coloured nri^ibouring clusters, occupy a 
5 relatively large proportions* ths pixels cf*fce image arcdeKnirdtobt; background. The 
hktugrajn for such a taetgtnjDd cokw cluate? is denoted by lim; 5C. Conversely. 
chiStat colours Much together with cJosery coloured raigfebtwriog clusters occupy only 
a cclatxvtily small pTOportiou of tiio pixds ofthc image a*e deemed to be foregrouni. A 
typical histogram shape for audi a fbjefirmrnd colour in rcnnsss>lcd. by line 37. By this 
10 analysis, obiter oofcuts can be allocated a defaxitC salfeqoy based on iho Likelihood thaf 
tbsy hib fijttt^OTttid colours. 

- However, colour mapping is not (he- only piwisss titfct is applied in order to dctcrrnip-s a 
satieucy image. In General, those regions which arc located towards the z$ges of the 
IS image may b^pcuaJised as ttay may belong to ohjech which arc not infy in fame. 

Further prorates, such *s paltera recognition may also be applied to die image! Thus* 
h search nmy he made to identify bodies or feces a* arxsnli of oomjwdu£ areas Yrtthin 
the jjaage against models held wi&in a modd library. 

20 

Figure 3F seborarficaUy iUnstrates a safieiujy imaga of Figure followiriB dK 
conclcuion of the one or more processes pCTfumiud in step 55. 

The saliency image & processed to subdivide it info a **a*alJ number of hxgz areas 
25 (typically recUulgics) vddcJi endoea die majority of the saJiency m die imagj^jas shown 
in Figure 5. Thus,, the selected areas enclose flu* bright legions of the salkncy image. 
One method of doing this is to form the sums of satfency pixei values along each ttw, 
and separately, down each column. Plotting these sums against the vertical and 
bcrizoni&J axes tespectivdy, ahowu die vertical and. horizoiWttl distributions of safiency. 
30 Thc*e can then be analysed to find the widest rami mum En either the vertical or 
horizontal aalkncy diatdbution. The imagA can then be split tow three parts at this 
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minimum. A first part compriae* n hoiizonraU or as tho case niay be vertical,, band 
' through -flic image having a width substantially coircspondiwB to thai of tho minimum. 
This past can lie ignored as non salient. This will then lefm? two parts of the image 
each- .vide of this miiJmtrm band which will conJain saliccyy (except in fhfv case where 

5 the minimvnn bsrwi is Jidjacent one- of the edges of the ima£c in which case there will 
only he cow non-empty or salient side). Theso parts can each be procetacd by the fierae 
felgoritLm. The past -with the widest smamuro can be split in an analogous m inner, 
discanUne the width of the minimum wd hence ^plistiiig that part into two smaller 
parts, This process can enntioue wftb each utagc qdilting the part about the best 

10 naurimuin wotfil one of the feltowng foraShtg conditions is reached: 

L No mmimiim can be found in may of the rcmainkig pints. Lc. no rainimum is 
found which is sufficiently wide and sufficiently low inadtancy. 

ri. The fraction of the total snftency or ftrj iruage which is outside of the retained 
b Jock reacts pnc?e predetermined limit, such ns o%_ 

15 

The result of this process is that a small set of pStoBflufar hlooku which enclose the 
inajor areas of satfancy of theimaBc are derived, as shown ha Flgate 5- 

Once features relevant to tlie imposition of the imago have been identified, that is itp 
20 to and including step 56, the saUeocy map can raw include ire'ioas of the image which 

aw defined as include regions and exclude legions Thus, considering: Figure 5A ffm 

girl hau hcec identified ai an "include" re^ira. ui*i has beea freiced fry a crop boundary 

60 which repitecnta the minimum boundary poisdhk: to include all of the girl therein. 

Sisai Jariy, the flowsn have been identified as an roehide region and have beesi mtW 
25 by ft txop boundaTy 61 repic^entiag its minnuion efup It4(tire4 to ti5d»dc the ft<wer&, 

FwShennoie, "must aachide" icgions iiave bees Identified and enclosed by WPP 

boundaries 64 and 66 respectively. 

Having Identified the mirdrourn crop boundary, it is then advantageous to identify the 
39 maximum Drop boundary. With regards to Pigare 5B„ one potanM nuKbra&n crop 
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boundary 65 1ms bftu CdentifiaL This crap boundary abuts the must exclude regions 64 
and 6B r but alBO abuls thi edge of ibe mns* include ru-gicn 61/ The boundary ale© 
exlrnda betwew the upper Bad lov.tr edges of the photograph. Ibis crop boundary 6ft 
repicscRte the mflximtrm crop boundary available to include the gjTl but to exclude the 
5 flfrtvers. Rowcrwr, an alternative crap boundary is available which include bolh die 
gid and the flowers. Thus, ob shuvm in Figure 50 a further ranjunjon crop boundary 70 
can be deJGned which includes both tht fikl Kid to; flowara (with pnrtM cxefasba of 
the flowers beirxg allowed because fhey are. so close to tbe edge), and a farther 
maximum crop boundary 72 has also been dnfiflad which extemis to the tipper and 
10 lower edges of tht photograph, to the teft band edfit, but abuts ih* must exuludo regions 
64 and 56 at fto 4ght hand edge thereof. 

Rc&rdng lo Figure 6A, control coremienoea at step 80 wbej* tbe sahenty map is 
analyEcd in Order to determine how many arena of interest eadst therdn. Tiros, If flic 

15 eatiejocy map sffu>ws N distinct areas of interest (for example areas of interest scpatatod 
by some area of nori-iutocfll as dttcftntned by some adoptively set threshold) possible- 
rjuiohnum cropping rectangle con be ^cnunilwl vdrkh contain: alternative combinations 
oftdwwft 1 odd N arcaa ofmlctc^ where The mintooto Mopping iccUrngfe contains a 
selected combat aiton of areas of interest and *xcfi>doa other areas. Thus this 

2f> corresponds to gtneratiori of mijjijjjiira cmrrpinL; rectangle 60, 61 and 70 in Figm-es 5A 
and SC. It ehcrtdd be noted that not all coti&binatibflS may be possible as obey paay not 
he contained utftfun & single red&ngio that csclndes one or mare of the non-selected 
areas. The maximum cropping mctantfo for the each szagte or eombinatkm of areas of 
interest is tire maximum rectangle which comabiS die areas of Merest bat excludes die 

25 Mrt-selected areas of interest. Tbns this cotrarwds to- rectangles S3 and 72 in Figures 
SBandSC. 

fiich roironnim ctoppbi s rcctengte 60, 61 and 70 sad its associated flwdmtHn cropping 
lunit (of whi.cSi oaly croppiag. limits 6% (Hid 72 arc ahovAi in Figures SB and 5C) ia them 
3D processed in torn. However, some initial sorting may reduce tho pRwesiiiag required. 
Ode of the composidojtal nilca may requite thai a large vrell ccrrirod tntccesfog area in 
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the image is deemed to be essential If we zpp'sy this mte than, only mminrum cropping 
boundary <S& and 70 are peimittetL. with the flowen -as defined hy crop boundary tfl 
betng excluded. The fotf. step is to select a first one of the mfflhrxam cropping 
Txiundarics 60 ami 70 as a potential croppirig amdirfate, together with its cropping 
5 limte. ITde process Jsperfbnued fil step 8J. From Jhereon, control is passed to aisp 83 
which seeks to identify possible edge locations &r each, of the edgea. 

The procedure at step 83 is explained more fully with, reference 1o Ffgwe 7- &ypposmg 
mat mmmiiim aod BTOrimttm crop rectangles haver been defined, and thai at now 

JO desired ti> fitxd the preaticyn of suitable crop boundaries between the midaium aW 
maximum limits. For Ac purpose of thla description, w& n» golug to locate "tee edge of 
one boundary: occurring to the left hand ride of the minimum crop rectangle. Given 
that das distal hueee can be considered as cenastittg of a plutttflty of columns, the left 
hand edge of the rnaximnni crop rectangle is located La cntamjs \vbertaslhs Ipft hand 

J5 edge of the minimum crop rectangle is located in colamnQ. Colunuis P and Q am not 
adjacent 

Sequentially oacb of the cnhirtmn between 1* and Q Is examined in torn in coder to 
generate a metric of how good tturt column would bz its a border of the topping 

20 rectangle. Thus, the metric is constnrcted such that da* areas or riowiy cbangtBg 
pixels along the column incur a low cost penally, wherert brighter area* or alternatively 
Fftpidiy changing colours* in a row of pliels achieve a high pcaalty sating, Forthctrnoxc, 
the rating may also be modified 5v5tij regards to the rnmdmiiy of that column to the 
minimum nod maximum crop bowidnxics, or indeed the pruntimity of that column to the 

25 edflscfflie picture. 

Ina preferred ombodtntont of the present invention, the edge quality metric i* & function 
of: 

a. Brightness. Tfaatfc daik edsea axe preferred and house menr only a low peaalty. 
30- b. Activity. Thai is the of the cobut difference* belv^ftregioas crossed by a 
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row or column is analysed, witfe low $m»s scoring a lower penalty^ 

c. saEcircy. That is the 6 cm of the saHeooy -values for pixels in the kw or column 
in fit) cued, with low saliency incurring a fewer penal ty. 

d Distance ftoto strung colour transitions parallel to, md oo the inside of; the 
5 COftllain or row being tested. The distance should not be too doss cor too far 

and a writfrted distance term is used to accowpLtefc this. This totter criteria is 
used to avoid cropping loo close to a feature, even if it is not part of the 
m i ninmm , cropping rectangle. 

10 These factors arc independently smoothed and nnarndfised before befog oorabiaod in 
order io fctrrQ n weighted sum to generate tb& edge quality metric as sfcowntn Figure 7. 

Thus for cacb one of the individual celsmxs, a penalty meas^rsmeut Is formed and the 
penalty mcasurEOKOt can then be plotted with respect to column thereby obtaojoag a 

15" penalty mtssucciiKyil profile The profile 90 can then be cxamiacd to drterrnjic thu 
position of nweuna Ifccrelll, such as broad minima 92 or flic diflzperminjpWM and 9$ 
Which are then deemed to bo potential cropping boundaries. This process cm be 
repeated for csch of ibe left, right* bottom sod (op crop bcnrnikxics indivjdatliy, and 
may be repeated on & iterative ha si a such that for a&mpte thotfc pixels in tbo coluFna 

20 which lie shrive the upper ejop fiinit ck* below the lower crop limit arc exported from 
the next ittereiion of the crop boundary, These candidate, crops can then be subject to 
farther constraints. In practice, there wilt be too many constraints to s unary ail of tho 
constraints aimojlaneously, steps 84 and 85 are gftea as as ampler; of the 
implementation of 5O0*e of tho conirttaints which ai&y be optionally utilised io the 

25 {denti£ea4Kui of crops irotn among <bs many ^adictete ciops which may hero been 
prcvJuoedatstep 83- Thus, at step 84, an attempt is imda to identify a horizrra line, and 
those crops which place the horizon a fi»ta the cd#> of the candidate csop are &.vonrod 
over those cropping possibilities that do not achieve this. Thus,, this corrc^onia to the 
imposition of the ^ndo of thirds* us respect of the horizon "Ene, 3imilariy> the "rale of 

30 thirds" can he inuodoced «t ep S5 to act on fee main feat tire cf interest Io place it % 
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of a dblaoec frrsn the cdfte «f the oiop - 

The final crop call also be conairaiaetl by the aspect rado (hot tte user has selected, 
Figaro 6B indicates fiio processing that may bo irapJemenlcd m <wder to sdect candidate 
5 crops bswd on their aajeel ratio. 

Control commences e£ step 300, whew a question fa asked as to wheOysr or not tite 
cropped, image 5s to have a facJ aspect ratio, Ifcb easora thar ft ia ptfaaiblt- thai a 
particular aspect ratio can bo specified cod enforced. In practice this, means lhat tvhea 
10 an aspect ratio is specified (control passes to step 102% a smaller number of otJjar 
constraints tv3J in £en<ra3 b*= rt^uired to comply spccitfy a crup candidate, than lu 
Ifce alternative case when 0,0 aspect iatio is required. Ia the ease of no explicit a^eci 
fistio rwrtmcmertl, U is tikeiy that an evaluation nde in &rcp 106 will penuSze 
tfdkolwasty thin aspect ratios. 

15 

tbv* & crop ettadiditle has bom Identified, it is then evaluated atr,tcpWby applying 
one or more ndea , Each role is j inplcmetitcd as a Jussristicd ty evaluated measure .on the 
hna#J- For example a metric 107 which measures how clopeapciniof interest inborn 
a ooe-Hhtrd line is shown ia figure & the Jfowtioaal position of the point in the 
20 candidate crop ia measured ia both the horizofSfll end vortical directions. The penalty 
for each direction Is determined from the tewistfcalfy dehammed graph shottn in 
Ksure£. The two mcawtcs penalty^, and ptmtfy'^ arc cotnbhuxf by the ru?o: 

25 if ffi2x(/wnaiiyw . penalty ^ ) > 0.75 
and 

30 if nw^penah>^ ,/>e»t% tof i ) -* G/75 
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Similar bown'itic measures ana used for oflter cufljpesiiiwial tdtos such as eliminating 
diptaudiotis dose tr> the edge ofihi BaiBO, miniunini edge quality, a rjjefetcocc for dart 
or low jyyfriry boundaries, and bo on. 

5 The combiaalkm of different nil* peaalties by a weighted enm allow? some rules to be 
considered as mons imported than others. Again, the welgljlingE era drtennined 
hfitrristioally. 

These arc many possible extesBicaa of the "basic scheme. For example, it would be 
10 possible for the role corobiaatiod weightings to be dynamically adjusted according to 
the overall type i>f faiag-j. For example, crop rectangle 60 with a singir area of interest 
containing a single far* looking straight towards tire camera may reduce the weighting 
for me mile of thirds, allowing a more ceaurally placed portrait to be preferred. 

15 Another possibility is for mi oddiiio*MJ penalty factor to be ecaemted faun srcp 81 
where some crop rectangles Are mtrinetcally preferred (U- given a low penalty) 
compared to othen. 

In Figure 6B f liic penalty Is evahiated as follow*. Ffest. a test 108 is rxsrftrtrtcd as to 
20 whiter or not the total crop penalty is less than a lowest previous total crop penalty. If 
to, <hw (he current orop candidate is itt*«icd <t (3cp 1 10 as the besi crop csttdkiafe so 
fin. If not,, then a test is performed at step 1. 12 as to whether or not there Are? mere 
constraint combinatlojiB left to test. If so, men the flowchart, loops back to step "IBti. 

25 If not; flie flow chftrt next .testa at step 114 if there are other arimmal. cropping 
rectangles left to test. If so, then the flow chart loops back to step S3. If not, the flow 
chsrt" shows tliai the best crop candidate is returned at step 1 16 as an output from the 
process. 



The idea behted Figures 6A. and €B h that all <*iii*hw!*>ijs of position arc generated 
and then cvaieaied in the later steps. This is a "generate and test" approach to 
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de&numing one or more suitable ctop boundaries, using a "cansmani set? dttetmined 
by mfoirauai and maxinnim croppmy rectangles. Figures 12a and 12t« iitasfrate 
GX2tti2$£sj of crops generated occordiiig 1o T iiD presort mverUian. 

5 Tho creeping selection process may be modified, to on embodiment of lie present 
invention, by giving the- user the ability to "tog" an object such thai it is included. Ons 
wy of manually tagging features is to use the camera viewflndwr as a pointing device 
(OS opposed to Us com/emioiiai ttso as both a pointrag and eomitosr&m device). In rhis 
use fta prime area of iiaterest is deliberately positioned in tie appnnriinale centre of the 
10 fraaac Wiien the irrogc ia Auto -cropped aocoidiiig to this invention, the region at ths 
centre the iraace is deemed to he essential to die cropped image and ja dms prevented 
from being cropped out, 

la another variant of the invention, tire camera may be initialised to identify certain 
15 coburs or texture us having a high. inJere-st At least two use models are possible haw. 
Quo simply involves Ihc identification of featnres of "natural" importance or iflkerenr 
interest: faces, tho overall shape of a person or object, and other expected oompositiojnal 
elements- Another is to provide additional dements or appendages ibr the- specific 
ptapOto of "T/Jgj^raii" to fcrco inrfasion of an object in the cropped image. The practical 
2* effect may be sfoiilar in ei&er case. For e^ample^ if a peraon is waring a blue tain 
jacket, then the camera may be pointed close tm at the bhwrahi jacket and flten capture- 
on Image of the- jacket. The cameta can then be programmed to process & captured 
image to assign Ural particular colour a hl£b jrtfaest metric. If a wide angle picture is 
then token of a sixain which the brae jacket appears, then tfaia area can be unsigned 
25 the highest cntwtst metric so (hat tie captured ira^e is autonimically cropped in svdh a 
way that too blue jacket ic retained, in the image. Thia ia particularly useful when, 
images axe captured of a crowd of people,, one of which tot photographer would like to 
make the main subject of the phologtecph. 

Figures 10 aed II allow in mom general terms lie concept behind ttspeoriveby the 
30 "constmiirt-baaed" and "generate and test" appmaches described above. In both cases, 
iiic Stirling £K>£tit. 1b *tt)c- id.tnti£co.ti<M?i o£" fboiuri^ r&fevtUiE to (bfr wduijtosj tin i\ of £ii& 
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image 200,300, for example as set out inthu steps up to an. iftCfocBnfi step 54 in figure 

The next step 202 is to determine Ac rderant Vuostoal&i set" for die identified. 
5 feaaaras* Hi* concept of a "confliaint eel" j'b a set of alternatives, only o»e of which 
should be cor^dcred at a time, EachataroauVe coceiati of one or more folly qieoia^d 
coratrtdrtte, far example features with some ^ftrrrcd vsiwe, lVflich are iben ennmciHted 

10 A simple example of a constraint set is ^'aapeti ratio"- These an? two altenafivoj, 
^pocfarir and 1aflj4sc&pe". Hie first attcnaavD {portrait) n^ght he defined by the 
consuabt 

(Xigh}- UftUBoilom-Tap) -0.75 

I? 

The sewsod atteitiirtive Qandscnpa) might be defined by fcs constraint: 
Gtlgto-Lqp) I (Bottom -Tvp)-\S1 

2ft With rcfcscoce io Figure 9, a more ccmsplcx consfrafait'aet could define o^Rphg limit 
alternative & for diffettm groupings of areas of toterca 184, 185 within live: maxSimmi 
boundary 390 of the original «$fcued brrafce. The number of atemaifvcs h\ tbia set is 
dstamdn&d hy trie anaryfuE; of the area e cf hicrail in the linage Suppese that two arras 
of interest l&t, 185 have been determined pisel cooidtaates; (SQ.E80) to (20ft, 

25 3flffJ and (330,50) to (600,240) w shqwi below. To afeex&jxrple, the lunate linage has a 
fop left coKJirfinale (0,0) and bottom right cnoidinate (640 t 4$G). 

His crop timifc on rarra ml act would coasist of larto altetutrvest 
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30 AJteriKttivc 1 (IcR-iusad area of interest 1 84 onfy) is defined by lbs constraints; 
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£ejt<30 
Top < ISO 



&ghi>2C0 FJtfit<150 
Bottom > 300 Bottom < 480 

Aliernativc 2 (righi-haoi aim of irttarafl 185 oo!y) is defined by th« constorintii: 

Ufi>200 Z^KMO 
Top>9 Top < 50 

10 £fc*r>600 Wtf<640 
£ofl<u»>24Q £&JStan<48l> 

Alternative 3 (iarilrjrie both areas of intcies* lEtf>) S 5 defined by tfaa wDstemls: 

T^p>0 7bp<50' 
K^Jk > TOO Rgk<64& 
Botiom>->QQ Bottom < 480 

20 The constraint set coficspi can be used to lcpiescqt mmy mutually wcctoilve sets of 
alternatives. Typical examples include: aspect ratio; allcnialive subject choices based 
on &c mtntmal crop roc tangle and maximal crop tests of various gronpifigR of areas of 
Mersfit; horizon placement oltemnUvvs {bottom third line or top third fine}; pohlt of 
ititercst placement (al each of the four Vie-thhcT intersection points or, for e$oij^ted 

25 itsau, aJoog ok or me four "one-ahirtr tmes); prat preferred edge phaanetrls for top, 
boKsm, lefl 4nd light edges. 

For each edge tapis a cajrairaJjtftt* ConstBtrog of aluanattve wnfica of distances thai 
are acecptabla on tte basis of an edge qualcty metric, 

30 



ITie efcflttrpios given above are all ^hard" oonstralnfe. Thai i%, tho condition axast be 
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tsstf mil there is no gradual penalty involved la dealing from (he condition- bi many 
cases it is desirable to implement constraints &e "•soft", that is, incurring an. increasing 
penally the further away the solution moves from the iocai optimum. An cxamplu is 
that poiitioning a horizon line exactly on the onwisrd Use is better fcapleinerited in a 
5 way that allows pl.iccmcait 3 gsfe way off the preebe o»e>*Mrf position, bat penalises 
ipcreasiiKg distance from the desirtd ou»4hinl location. 

The optimisation problem can easily be set K> taelwdc this. Fw exampje. by changing the 
condition: 

W 

to 



wissia a/ and c3 are positive penalty terms which contribute to aji overall penally 
fflnction to be opUmised, JypicaSy as atveigtued sum of contributions sucb as.„ 

penabyclel * e2te2 + ..... 



20 The fl£H step 206 is to pick the next cenabhxalion of constraints, one alternative fjom 
each wmatraint set Many conahifltrtiaas can he hmnediatcly ewnteded or simplified as 
titty are precluded by some of the tftbw constraints. For raanipb, choice of a pardcatar 
propping Bunts «Um native will limit which points of interest can be considered aa some 
may be outside those cropping iiau^. 

25 

The optimsl etop tarclidatc for the aadtjni constraints can then, he determined at step 
208. The ctHisfraiiite we have get up are ccmrdnatmns of simple linear conditions. 
These can be effectively solved by linear programming methods which find the location 
for Hi?, bottom, left and right boundaries of the crop which meet ell die hard constraints 
30 and satisfy the soft constraints in such « way Otat the ovwaJfpKnally is rrdnirxuBed. 



(59) 



JP 2004-520735 A 2004.7.8 



28 

Dcpcddiflg on ihc precise set of conssramt combinations being solved, there may be a 
number of satiations- Ideally, there is a single optimal solution. 

However. th«e may ba to solution, This would he the case If some conaaaims trere 
5 cofllradictoiy. Forejcampie, iTiherearc two points of interest AandBwifcero Aistothe 
left of 3, and a einnbinBtiua of constraint Owl attempts to place A iiear die? r^tf Jiand 
oo&tnsrf Hue and E near the left-hand one-third line, rhcri there ig deacly no sofotioa- 
Tbe aie&od in step 206 orf selecting sets of cottstomte to solve should ideally be 
implemented in totcb ft wiyas to eliminate these, tdtua&ms. 

10 

There may he muHiple sohmons of tonally low p&nslly score. In tins case have a 
number of aUecnatlves. 0;i& is to pick a sokitivn at rasdom within die space of multiple 
soJutfoDa- Another it to tighten «2ie comnuhif^ fox aannplc bytutring <x>o « mow soft 
cnastmmiB into hard co&steddfcL Qirifonaiiy. in cfep 210 it is possible touaea ricfief 
1$ evaluation irclric to Bonecfite a set of alternatives within the* space of squally jwctpfsblo 
solutions and select these on the basics of toe refined evaluation metric. This optional 
slep msy, fot exaamle, be ft "^ctUanUe wad testT method. Many variant} of this mo 
possible, 

2d The linear solution b q practical method that Trarfcs well for automated, cropping 
because conslrautt sots can bfc fcttned thai reprasrot nSlemative plausible, chokes. 
. Treating each combination indc^idenuy and hence finding different locally optimal 
solutions* is a useftd way of aatcratTng good rdlefltanvca ior a user. Nmvlincar 
optimisation mcUiods frequently siitVer from problems with localfy Optima] whrtiora 

25 being contused foe gtobatty opthaal solutions. An uqwei understanding oftfc© search 
space allows this tcclicique to rircumvent such problems in a relatively intelligent 



Although linear programming is one method thai may he used in step 208, It docs 
30 impose limitations op the way (he constraints a» dciincd. Diner cpthviisHtjcri 
techniques couM be «sed within the some basic framework of local optimisation whfain 
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jj. Faibapa.cc defineti by lie f^toiee of cosDStfB&tfs &om -constraint sets. 

If ell possible conubaints and eveJuatinn criteria, aie encoded as hard or soft cxj^Mnm 
which can. bs optimised in stop 2DS, than slop 210 may be bypassed. How Over, -if step 
5 20S is achieved by liriear programming thta sonic- of the conpfrsinlfl stay be poorly 
uppccsiniftKd or omitted. A more accurate cvftloatio.i of ftps eolation generated by step 
208 can be obtained afterwards hi step 210, A, more refined implementation might use 
the nj^jHruriTTifltc solution, from <4ep ?.0S ;r) lk; atmt point for a "gfciwralc hikI tesT based 
kfcal aptiOMsation using the tworcdcfciaBd ewdundon. n&otrics. 

10 

Au. exfldpte o f constrakits thai c an be only ftpproxircMed with & I siicr-r rcpresen tattcri , is 
it* edge quality rottriefc The tnn edge quality can only really be assessed when the 
lirnte of 'die edge gre known- For oxamplo, die trite relative qxralrfy of alternative left 
edge locaSqjcis h dependent on the lop and bottom limits. A. narrow choke of top and 

35 bottom way ex etude features in the im^ga that twold otherwise adversely ar3bct rhB Left 
edge qnalfty. lhis type of hrterdependeacy cannoi bs modelled wish a linear jjystem 
Tiifl best that can be done is Stat whbin. step 204, having selected mhOTWrci and 
" n nu c Bi u n o cropping 1 unite, ths edge quality metrics ace recalculated using, for example, 
the maanminn cropping limits;, to generals a specific set of afaeinative constraint for 

20 each edge constraint aot Tbasesperffic eoDstadnts Jo* used *hile tltf> particular choke . 
of cropping Bunts is bt^ngcoftard&cd. 

Anatbcr type of consmtint that cannoi be. modelled tinently is one itu«Sv£eig ratios of 
areas in the imago, for cxycopfe, the rslaJive area of & boring region within the crop 
25 bound eay. Ci&iux^ this will be nonlinear as the area is n muMpUcatioruof terms 
involving tfco hozizurual and vertical crop locations. 

In any event, once the crop candidate lias been, evaluated* fins is recorded at step 212, 
olrog with its penalty score. 

30 



Pram step 212 control u passed \o step 214 wi*eieat-st is pcribnned *a to whiter or 
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not all- constraint set ctmibiuali-yas have been, tested, if act, the -flowchart loops hack to 
step 206. If so, tins there are many possibilities for deciding iwhai to do with fee 
results. Figure 10 shows josJ caw exarspie, in which the crop candidate ana sorted in 
order of penalty score at step 216", and than a Dumber of those having the lowest penalty 
5 scores are presented ar step 218 to a user of the system. 

In addition to selecting, ftoni a set nf alternatives, a user may wish to suggest 
improvements to She selected alternative. This could be achieved by simple commands 
such as p more° or. "loss*"- hi swh, cases the system could define a new minirmirn. crop 

10 rectangle and a new maximum crop limits, bayed cm the selected crop and the 
alternatives which the user rejected. For exaojpje, if th= to er requested "more" tlKJi (he 
selected crop becomes the new mmlmicn crop rectangle and the smallest nWselected 
crop wbVch exceeds toe size of the selected crophcoclnes the new maximum crop limit 
Ha system can then le-oonsWer alternative edge placements within these limits to 

IS genera!* a new se< of afceruadraj to preser* to the user. Repeated esc cf this form of 
interaction, can allow the user to hrttratrivcly explore toe space of alternatives which 
best meet the system^ criteria torcontooaiiiotfKdly acceptable crops. 

An example ofautrtoet apptoscb iwisW bo to cn&ue that oas from each of the possible 
2tJ ahemtttivc crop laeha was used. 

In acme s^pUcatkusG of the invention, a user may not be involved, and the crop inay be 
jft»!ly automatic. 

25 Figote 11 is a general example of a "generate and test" method, in which certain 
coaatraiuta are defined as hard, equality constraints and are used to define a crop 
candidate withtfot any attempt at ophnmatfioti. I2ach brad ccmstnabit is a local optimmn 
location for a particular criterion. 

30 For case of comparison, steps in Figure I! that coree^pofw! with, those in Pigine> 10 are 
given, a reference numeral incremented by 100. 
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Olfte the relevant constraint $cls for tlic featwes liave been determi&ed 300, these ate 
cnurneratcd at 304, and a group of constraint seta £s selected £t 305 as ^driving 
CQHStfiitLt 9£&". Tries; cunsfeUnt gets ace such that ivhan groups of coratmkte are 
5 formed, ooe feum cadi driving constraint eet, a enrp candidate is futty s-prciGed. 

A simple crumple is for the group of driving corrsfcrainla to consist uf 1 be Cattstfaini seta 
for top, bottom, felt and right locations, where each of fttcso constraints is a candidate 
edge position dstczmrnsi from the edge quality rneiric. 

10 

So, for example, Ibe left edge constraint &crt might te*. 



Left™ 5 
IS Lefl^lSO 

Analogous onnsiraiat a&s would exist for poasibfe Right; Top and Bo/taw candidates. 



la tht example discussed above, the driving constraints are edge constraints such as 
20 fhc.ee, combined with the cropp ing limits for various wmbiGfiu'ons of areas of interest 
(i.e. ruin imam Crop rectangles and cropping limits). 

An nltemntive group of driving constraint sets .might be ihrcc enumerated edge location 
constraint sets and an aspect ratio constraint set. 



The driving constraint sets determine which oat of all pmsfoie crop rectangles are 
"generated". In the least ralcUigoat possible implementation, fdJ possible left, right, tup 
and rKTttorn locations are generated, Thda, hevrever, mcrcascs wraputsiionai cfihit 

30 la step 306,. the- nest cotncnrMtton of driving oonstrarats so selected, otic from each 
driving contfxaini set. 
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The o^tenrunatfon of flic current crop .candidate in step 3QS is trivial, as tbcre is only 
on= possible solution to rhe= driviog constraints by definition. 

S In step 310, all soft corts&ntata me evdruatcd, Hcxe 7 zn unspecified colleoion of soft 
evahmticci cntsna are combined. 

0»ee U*s crop candidate baa been evaluated, Siis rs recorded at step along whfc.iis 
penalty score, 

10 

Uttsi a test is peffbrrocd at step 314 as to wrtatfcej- or riot aJl driving constraint set 
combinations have been tasted. If iini, }lic flowhart loops back to stfigp 306. The 
flowchart only loops round flvs driv*fc£ constomita aa theses fully specify the Candidate ' 
crop reccing] e? to be considered. 

XS 

if ail driving constraint sets iavs been ensealed, then thnye arc many possibOitiK for 
1 deciding wiot to do with the resufts. Frgrzrs It shows just one eKampre, in wfuch the 
crop candidates are sorted m order of penally score at step 316, and then a number of 
these having the lowest penalty scorea ere presented to a user of the system at step 318. 

20 

The invectioa provides various advantages in jmttonatlng or semi-aitoraatf&g tbr? 
capture of cropped images with impro\-ad selection and coxaposihon of subject tanitcr. 
The invention also rcduoea tbu ef&rt on the part of the photographer to tabs 
photographs with good nod appropriate conrposMnn. thereby matching the potential of 
IS an electronic camera to caphtee a large number of images quJckiy. 

It is thus possible to pro vi do a method o£ and apparatus for analysing an image whkh: 

a. buplernents a aet of metrics computed from features of the image, each 
embodying a composrtinnal rule; 

30 b. Utilises dynamic ecc'iSl'Jtx laechanisrrw which liecidtj which ofth« wjrajwsitioBal • 
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rules arc appropriate for a jgive*! image on dm baste of the cwneoi fifths image; 

and 

e. laeorpoyates a mochsmism wldrh optimises the pasiiion of the crop boundary in 
accordance with the approbate; compOBrtional rotes. 

5 

A largo number of coaiphsiun-rtal rules can ha umdemertfed under tiw !>ubeinc, a 
non^ uniting and noowxluustrvu list of: which includes: 

* Identifying distractions which shotild be excluded frano the scene. 

* Jdcatifying horiug areas which should be reduced fas size. 

10 * Identifying a horizon hue and ensuring that it i& not «>(aced centrally Or close to 
tfactou oi bottom of the image. 

'• identifying impotimii subjects, and arranging their pla cement in the output crop. 
This may, &r example involve use of ma we!! known "rule oflHrds*\ 

* Identifying mid tip lo possible subjects and gnaetaJhsg alternate crops contakmg 
15 dhTcrent Dcrabmactona of subjeefs. 

* Identify biff die approximate direction in which one or mom persons 5s or are 
looking, rutd, seeking to provide more space on the side of this image that the or 
eaeh person is facing. This rtilo jnay not be satisfied if two or mow people aro 
facing In fhlfetont directions. 

20 * Ensuring that bou»3ati«t form aattttal 6*mea where pojEHbte, for example fey 
selecting tetati vtiy dark intotrvo arena hi the image as mean m. which to plana 
bound arics. 

+ Erji>urin£ that sttor.gty rxwlj-totrng edges arc not placed at or dose to the edge of 



Eneufing tiurt thin objects pointing towards die edge of die franic at* either 
compJeldy in the feme, with some background border, or altstnatively arc 
decisively cropped. Examples of such objects include mitfrttetched turns or 
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♦ Cosuring that people 2x0 crapped, if ivrttfjisfuj-, t.t points which uk Liiown to bs 
pleasing. Tote typically avoids stopping at joints such, as ankles, waists, elbows 
orthekneck. 

* Constraining mo aspect ratfo of output crops to bo a raru2& of generally 
<*c&spttd conventional jsoiMJitiosw, optionally vdilch may be fixed to * desired 
aspect ratio. 

m practice, not cvety rule applies to every image FwitiennOrc, eome rules ttiey be 
contradictory and ftaice may bo combined in alighted mtuurar in ardor to define a 
potential crop jncurrittg.lhfi leaat penalty under the ruSea. However this invention 
provides a set of implemented rules which arc tested against fhs image to sec if, 
where, they ar= reJevjui?. A single rale nay be sppJicd muhjptc. times at the snipe 
image; for example removing a number of dlraactioos, 7\Aa is on aspect of fre 
invesrtfoniftijicb contest* \tfUilbocurrentry known artin whwh r>picalJy ordy- onorulc 
is used to isolate a wnfifc subject area and place this centrally tvMjIb a ftamfl or within 
some Actively small fiscd st?x border. 



E should also be noted that, because a plurality of composMoTud rules are implemented, 
20 it is typically not possible to &Uy satisfy all the compositional rules which apply to an 

image sEKzhaneausIy. A m.o&m is therefore required to identify and foan 011 & or laore 

relatively fiuodTOnrprosniscs. The present Invention allows that some or all of .he rubs 

Tvhicn arc found to apply to an Image can bo ttntoina*icaily formulated atj constraints. 

Various automated methods arc then pcasftfe to find locally optimal ways of satisfying 
25 a set of conflicting constraiirts, Another shjmfieam arid important feature of the 

invention h that each alternative local optimisation may itself be a snrfeiM e crop result. 

This contests with the prior art. TTias the present invroiion is uble to Bea«atft & small 

number of alternative crops from which a user can select 



30 



In specific irapkswntafioa of ihe invention described bereinaboro, compositional rules 
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were embodied in » number of Jcey fejt&Bea of*e processing steps. In particular: 

i. Satieiicy tug, generation wherein rules for identifying areas Kfrely to attract die 
aiteiitlon if a humor* viewer weie applied, 

ii. Rules for the UMifieattoQ and elaninatinn of distractions at fhc edge of (lie 

5 



iii, lodes for identifying diffident comttnations <?f subjects for alteqiattK 



iv. Generation of edge quality ineirfes, iaolcdma rules for selecting locations 
appropriate for protf si&tt of tho cropping edge locations. 

10 v, Eoumng n^tnghted sum combination of crop quidiiy medics and implenreoling 
■ rules for wunparativeiy assessing aJteoiaiive possible ontpor crops. 



The pw&on skilled in tbo art will, of course, ascertidn fiotn the above tenciiiti»5 that the 
general principles ofHus invention, lend tbemsel?es to afeematfve iraptoaystfaliaos. 

15 
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claims 

h An efectr»mip image processor for cropping mi electronic image, the imago 
processor adapted to: 

a) process the electronic image to identic 009 ur more featnres rel cvwit to the 
conrposjtion of the riecteoruc image; 

b) select at least oris compositional rub Jrom a phiroJhy of prtdetexmidad 
compositional roles:, based on th-: relevance of ibe compositional fok(a) to one or more 
nf rtiA identified tenturcs; and 

-c) determine occ or more euiiable crop boandarieft by applying the selected 
ooinjw hi rlurw 1 rule*. 

2. An electronic image processor aa claimed in ckrim 1, therein the image 
piocccsoT includes at iswt on* of firmware and Boftwnro for processing an image, 

3. Art efeckorria hoflfre proeesBOT s» claimed in claim 1. wheffcirt die processor is 
arranged to analyse the image in order io produce asalieroj- map. 

4. An ciectconic imeg& processor as claimed hi claim 3, wherein die image 
comprise* a piutatfty of pixels and (he mage processor is atreagec to dtetcr aamUar 
pixels kd th* wt^itstotegiortt 4md due regions yre analysed to determine a measure of 
unusuamcst Tvilhrn the inrag^ 

5. An decfroiur. image processor re dairtiwl kctaEm 4, juwhii^reeioEs whlciurre 
ttsoBufll ate mted tw more salient than regions \\hi& occur more commonly. 

6. An electronic image processor aa claimed In claim 3, wberein the image 
processor searches the image fox (drapes which according to a predetrmnrtcd rule arc 
regarded a? xtisvwt features, such features iDcJuding &ces of people. 
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7. An electronic image processor as claimed ifi claim 3, wherein the image 
processor searches Are image for diapes which according to a prsdetenriLned rule arc 
regarded as rdevatn featured sack features mcladlog tagged objects. 

Aft electronic image processor as clainccd in claim U ra which crop limits are 
^cnfTfUKJ, and then suitable candidate crop boundaries betvyeso the crop limfts are 
evaluated by generating a measure at quality for each candidate cicp boundary in 
accordance with at least ose quality metric, and a reduced number of crop candidates 
art selected as output candidates based! on the analysis, 

ft, An electronic image processing device comjufeuig a data processor anangcd: 

i) if* firmfyfce. no im&acio dctemiitw at least ©re oonip optionally eigniEcaii- festare 
within an electronic iKWigc: 

2} to npply Ht least one compositing mic to the fenrnre?; to identify threw fisrrnrea 
which are- to be iicSu&d in a cropped linage; 

3) to generate crop Huftft dining the. bowxbries of crops pspoeialed wilt at least 
one feature to ha included in ib& iruage, sad then; 

A) to generate and test candidate crops wifhiu the bewndenes to identify those, 
which in accordance with a qitaiity me trie, represent acceptable crops, 

10, An electronic image processing device as claimed hi claim 9, wherein the 
candidate crops are constrained by aspect ratio- 

11. An olecfrouic itsiage processor, areanged to process an dectrowic inonge to 
identify compcvaiionafiy sign W cant features wham the image, to son the features 
according to at least one compositional rule into those features which aro fe> be included 
within cropped versjoas of the iraage and these features whioh are to be excluded from 
cropped version* of the image* aofl Hico to Kirtorriatically generate a candidate crop, 
wiicrcin the candidate crop k not maiely the swaltest crop that encloses a feature wllich 
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12. An electronic image processor as claimed ra claim U, ismeifcm the image 
processor generates a plurality of candidate crops, 

1 3. An electronic image processor as dfcunacd in cU^iu 11, whereia sietabl& crop 
boundaries ore Meottikd by applying a. ptorttlity of constraints fa the imagc> mo 
coxtf^rnvats gcneraicd from, {east one compositional ntfe, 

J4- An electronic image processor arranged to analyse an imagD to identify 
cornpositioiialty sigmfiocnt taatuas thereto; to apply at least one compa«ti:o« ro!t so aa 
to generate a plarality of constraints; to eppty tho constraints to the imago Sn getterrtea 
plurality of candidate crops each one having a measurement as^ocjtted t3jeiewi£h 
indie alirrg tlw. extent to whtch the constraints are violated t and to present at least otic of 
mo better candidate cropp to a user. 

15. An electronic camera system for capturing as electranio irango of a scene, 
comprising, a detector array for capturing the electronic Image, an optical imaging 
system arranged to image the scene onto the detector array, and an image processor tor 
cropping Una cirpEurwl image, wfcerain the image processor ii as claimed in delta I. 

16. An cJer^rooic camera as claimed in Claim 1S» in vdUcit the earners, aiohrdea 
means by wjrieh a t&x of the camera may tag one or imwe features relevant to the 
compositioji of t£»eIecfroflic irnqge* said tagged twrturc(sO- ffien being associated with a 
compositional rule thai includes said tagged fcawrefs) ia Aw cropped image, 

17. A method Of using an electronic image prtwessfng device for cropping an 
electronic image* mo imago processing device compnshig an Image processing means, 
-he imago processing means including at feast one of an electronic processor, firmware 
and software fox processing foe electronic image, wherein die method comprises ths 
Siepsof using Ae image pi Dccssmg. means to: 
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i) piO«S£ lh.& electronic imago to identify at least one feature relevant 1t» the. 
composition of the electronic imags each such feature occupying a sub-regin& of the 
eluctronlc Jma&e; 

ii) select el least one compositional rule tranv a s^t comprising ft plurality of 
predetermined compositional rales, b«Kd oo the relevance of (hft composhioflal tulefe) 
to fft I,fciLf>t oc^& of Ait idcjitEJSoti festutit^ SEidE, 

Hi) determine at least one awtablecwm boraasipy by ^pplyiag an* or more of 6c 
selected GOiupo siikvnal cults . 

18. A tiaettiod ret cfemusd m Oauri L7, in which each identified feature has at least 
one composition ally sl^atficamt property from amongst a phiralilY of tfiffeient 
ptedelatfmited ccanpositianal properties, and at least one ccmpceitio-nal rale Is e elected 
from a act of predcCeaained composil}o0ol ndes> based on (iie relevance of the 
compositiosal rulers) to tue compraitionaUy- significant properties of one or mute of die 
identified featurea. 

19. A method as claimed in Claim 17 is Which step iil> comprises tta sicpa of: 

iv) generating a plurality of alternative candidate: crop boundaries; 

v> generating a nleasnto of eomDorition quality for each, alts-native candidate crap 
boundary by mang a set of piclare rampodtion metric; and 

vi) detecting a reduced osunber of crop caadktefea having a relatively high measure of 
compositioH quality. 

20. A method aa churned an Claim 17, ft which step <iti) comprises the steps o£ 

vit) implementing the compositional rules as constraints mat may be satisfied to a 
greater or lesser degree, each coasteaat having an associated cost functtow that 
increases thf leas vi<$\ ths r«iirtrainl is satisfied. 

via) defining an overall cost metric as auction of crop COliidirwtos in the image, by 
forming a combination of the separate cost fliuctuMM associated -svith each individual 
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cxmslrrfirU; 

iK> applying -no optimisation nntfkod to find at kast one best crop locations by 
finding minima in ths ovcr&tf cosl metric and 

x) selecting: asr output a rcdnced aaraber of crop caadMates for said best crop 

21. Ait^nd as claimed in CLum 19, m which only cat drop candidate a selected . 

22. A method as claimed In claim 20, in rtfacfi only one crop candidate is selected 

23. A method as claimed in claim 17, in -which tt*e cfop bpondary of the cruppsd 
fowge has a, ptemUty of crop edges, mid the steps of too method ate performed 
aspap tery for each crop edge to geijuatc 4?* crop 1 >:*o» >da.ry. 

24. A method ili daimed m doim {7, in "which 

- step 03 ineiades irfcr.ufyinjr. features which oonstitutB noo>DTCriappirtg segmented 
regions of interest 

- stqs (ii'c) is achieved by first selecting. akemativE drvMona of said identified region of 
interest Into those which wiB bo included by the «op boundary arid 3xh» vfiiich will be 
excluded by the crop boundary;. 

- each such alternative division of regions of interest is used to determine an inner crop 
boundary in nit sud. an outer crop boundary limit such list the inner crop boundary limit 
is the smallest boundary which circumscribes (he regions of interest to bo included and 
the outer crop hoaudary limit is the largest baimdary which exclude? the rcgWtW of 
interest to be excluded: and 

- said one or more suitable crop boundaries atte detetmmed between the limits of fee 
inner and outer crop boundary limits for each of said alternative divisions of regions of 

25. A method s* clabnrti b\ Claim 24 in wliiofx swi ireoer crop boundary limit and 
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aaid older crop bmin.rtjHy limit Ur.3 a pluialily cf edges and one or rmjeo suitable crop 
boundaries are determined by cvalaatfiig an edge quality ttiOric far encb possialc tdgp 
placement between the inner said outer crop boundary liciuts, the aoiootfotj of best crop 
boundary being al Jcast in part dependent oc file separate «dgc quality metric*, of each 
of the boundary's edges. 

26. A method a3 claimed in claim. \1 t in which one or n»ore of said suitable crop 
botimlaries are n«s*ntad to ft user of the device for maaual selection by foe «s« for 
chopping of the eleetroafc feoagfe 

27. A method as claimed in Claim 17, .in which in step <I) blank or uninteresting 
aiaas are detected as some of ibe features ratevatu to the conrpoaitj on and in steps <ji) 
and (iii) one or marc of the compos itfonni mtes cause the image, to be cropped 

■ 2Ccnrd>jvg to the fln^ngcnenl of said blatk or iKrintetesting areas relative to other 
identified fosrtws relevant to tilt composition of the efaolnHac image. 

2S. A m«W as cteimei! in claim 17, iti wtoh the image is a oofoar image, and at 
fess< one feature is identified by Mgmeartmg the iaa^ into areas of wtailar tolonr 
and'or texture. 

29. A method as claimed in claim 17, m vtadb in step (i) features Talcvmit to th» 
cwnposifion of the *te4nmic nnffgeaxc identified by a process of; 

- segmenting (he imager into regions; 

- denoting certain, rcgions-remon; salient than others; 

- groaping salient regions iflto forger regions separated by relatively ncmrsalient regions; 

- identifying sidd grcaps of wtlejot regions as features wfth n region of interest property. 

SO. A method as otabned bi Claim 29, in vrftich the image is segmented into regions 
by segmenting the image into homogeoons aBb-wess, the theasorte of homograjcity 
hemp based on -wine coodrataticn of cofonr, irtfenaity and tcxiorc 
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3!. A melt ad as claimed in Claim 29, in which cc,1«J,t piyiiu arc denoted as raare 
salient than others byelta^thignigh snttenot to Ptgioaaoa Hit betsis«f at least oik o£ 

* rdativt imiKFiaUicsfi of M teast out of the cnloui, intensity and texture of the tegkn. to 
<rffoer erijaif nt fCtiilofUv-; aitd/or 

- relative unusualiieas of at least one of U» colour, intcasiiy and testes of the region 
relative to a substantial portion of the image. 

32. A m«:J3od os claimed in. claim 17, in -*yfcich tha cc;u;i>o-sfue<nal rule for features 
identified at the centre of the image is to indole such features in thu cropped image. 

33. A method as claimed ip elnjm J 7, further curonriahig a Step (G) prior to sfcp {i) 
of dctcjpoim&g, independently of capturo of <he efectionk; image, spwific feztwea qt 
objects of interest, and wherein step © farther comprises trtteaqrted identification of 
said specific features or objects of interest sad designation of EUccescfuHy identified 
specific ffeatare» or objects of interest ss fc^es relevant to the composition: of the 
electro mo image. 

34. A. method as claimed in claim 33, wherein step (0) comprises tdentificatiftn of 
specific features or objects prior to capture of fho electrode iraaga 

35. A. method as claimed ju claim 33 , vdsciiem rfep (0) comprises prevision of at 
leapt one identifier tag asd of coks foe kkmfifieaiion ol identifier tags, and wherein step 
ffl comprises use of stdd rotes for ideudficatkm of said tdeatifier tags. 

36. A method as ctahncd in claim 35, whereua said identifier rage are infra-red 



37- A method of cirip;>mg on elevtruate image, the method cCmprLrfpg; the steps of: 
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i) automatically analysing the Imagt to kknfifj at least one feature thettan which 
» signiJ5cani to the composition ofthe jaiage-, 

n) automatically associating at lem bjk oomposiawi^ rule Willi toi identified 
feature; and 

hi) gracradn^ a crop boundary wBicti inohufea at lead: uuc feature Monrified as a 
fcatuJu to be iccjcdcd willymb,-: unago, the crop bouwbry selected lobs Jarger than ifte 
miotmum towl&iy reqi^d to enclose the f tatter. 

38. A method of cfopph:^ tsi. ek cltonfc ima^ff, comprising the tftcps of; 

i> processingthc isaage to identify at leas! oao distinst loaUtfc wilhin the ixaasc. 

n) seiec-riBg at least two compositional wles rdevant'tothoal least one. feertnrc,1he 
MMuposftiraial mica detecciaSng the posstbte jndtision and placement of the feature 
within a cropped version Of theimsge; and 

ill) detazniniog at enc crop boundary for a cropped version of the image in 
accordance with thv g^tectcd cotinwaitiDaal rules. 

39. A method us counted in ciahn 38, tn which features whdea are to be included in 
the cropped votetofl of the knagc arc identified, and the Spatial extent of these features 
la used to define a routimum cropping limit and vrfjereia a maximum cropping limit is 
also (fetetxrined and n test i& mada of candidate crop bo*3idariua bounded by iho 
rzunirnum and maximum ftoppiog limits to deteimifte Ijow well the candidate crop 
boundaries confimn to (be selected coinpcwiuona] rules. 



41). A mothnd as claimed in cSahn 3 EL, in vdnch tlie selected ccittpositlrmDJ rales are 
used to define a set of constraints, and then candidate crop boundaries are detc.roiined in 
accordance Vtiih the ccnstrointE. 
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41. A method as claimed in claim 40, in -w&cU the cvusU aiara or*- impkownwd as 
soft coxistftiints, such that the decree of non compliance wilh each constraint Is wk.i1 m 
form a penalty rosasuremcnt 

42. A method as clmmed in claim 41, in which fee crop with tho lowest penally 
associated fliotowith is Pooled for pfesmiatlcn to & osejr. 

41 A method as claimed in claim 42, bj which a user can. «mpo>» fjwsir choices into 
the compositional rotes. 

44. An «l«rtrwttc imaging jxocossioe; 4cw=e for cropping an dectronio imago, 
substii&tial]y a* lutein dttctftcd, witi; jefewnc* to or as showa id riic accompanying 

45. An. decttotiic camera system for captnttaa an elccttoufc image of a 
substantially as hflrtks described, valh reference to or a* shown in tiw arawjopawpfpg 
drawing. 

46. A method of using an electrode image prooessmg device for cropping m 
e2«tmnic image, subatenti^ly as b«<riadc3cHbcd, wim inference to or HS iiiowil hi thb 
Rgcompanj'mg drawing. 

47. An ckctnxuc iinafle |>ft>e«sot as claimed in duim 2, in wliich a plurality of sets of 
crop itmrts ere generated and evaluated, each set owBprising a mmiirann aroppmg Emit 
and 0- lyjiximum cropping limit. 
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Convert captured image 
to YCC colour space format ~J 
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Btur image 



32--, Form new "seed areas" that 
vf have smooth colour and 
intensify 
— . — ■ 
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/-Area Growing 
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Grow "seed areas" by 
adding areas with the srmflar 
colour and intensity 
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Merge areas sparated 
by weak adges 
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38~a Merge areas with similar 
mean colours 



V Area Merging 
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ether areas if below a 
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[ From Rg. 2A | 
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Cluster colours 



Assign an Interest metric 
to each region 
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Determine for each region one or 
more compositionaHy signrflcant 
properties from amongst a plurality 55 
of different possible such properties, _/ 
depending on the average colour, "~ 
Intensity, location in the captured 
image and fnterest metrics within 
the regions 



Select compositional rules 

appropriate io tne 
determined carnpositionelJy 
significant properties 



IT 



56 



Determine one or more \ ^S7 
suitabFe crop boundaries by J 
apprylng one or mora of tho 
sdected compositional rules 



Fig. 2B 
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/ Hgs 3a and 3b up to andl 
\ Including Step 54 J 



Identify features relevant to the 
composition oFlne electronic 
image using a combination of 
one or more interest metrics 
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{Fig. 2b Steps 55, 56, 57} 



Form one or more minimal cropping 
rectangles (MCRs), together with 
cropping limits 



SeJect first MCR as current 
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Identify a set of possible crop edge 
locations for each side 
{Edge constraints) 
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Identify horizon line if possibte {1/3 height constraint) 
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identify principle interest points in each sma 
of interest m the MGR. (Each pt gives a 1/3 /" 
heiaht constraint and a 1/3 width consiraint) [~ / 
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Pick next combination 

Of constraints to 
deJormtae fixed aspect 
ratio crop candidate 



Pick next combination 

of constraints fo 
determine unspecified 
aspect ratio crop candidate 



Evaiuate current crop candidate by: 

A) Testing each evaluation rute to generate a rute 
penaSty score in range 0,. A 

B) Forming a weighted sum of the n^te penalties 

to generate a total penalty for the crop candidate 
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Identify features relevant to composition j-200 



'Determine relevant constraint sets forfeitures J" 



r 202 



2 04-^ Enumerate the possibte alternatives 
In each constraint set 



{Rg. 3b 
Steps 55 fc 56, 57} 
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Pick next combination of constraints. 
H one alternative from each constraint set 



Determine the optimal crop eaniftdate tor the current constraints, 
(I.e. The solution with the minimum penalty score 



Optionally re-eve luate the pensfty for 
the crop using a wider set of evaluation metrics 

1 

A 




Return the N crops with the lowest 
penalties to the usor 



Fig. 10 
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{Fig ^|-. i DetetTn ^ ne rei&vant constraint sete for features | /~ 
steps 55- 56, 57} | 

j Enumerate the possible alternatives in each constraint set \ f 

V ' " 1 ■■■ ■ ■■■■ ^ ■ ■ ....... — , 

j Select a group of constraint sets as driving constraint sets \J 
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3G6~> pidt n& xt combination of driving constraints, 
one from each driving constraints©! 
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l | Determine current crop candidate 

{ 



EvaEuate current crop candidate by: 

A) Testing each evaluation rule to generate a rule 
penalty scorn In rang© 0...1 

B) Forming a weighted sum of the rule penalties 

to generate a totaJ penalty for the crop candidate 
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| Record the current crop ooordmates and its penalty score 

r 314 

[Sort alternative crops to order of penalty score^| j" 
| Return the K crops With the lowest penalties to the user[.j"~ 
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